Draft Universal Mulit-Eliciting Activity Style Creativity Rubric

This document defines a suggested single rubric for assessing creativity across all MCSI universal rubric categories. It is used by every exercise where creativity is assessed, so that one rubric applies to hundreds of exercises without per-exercise creativity definitions. exercise and category specs reference this rubric and, when needed, supply only evidence_location hints (where to look for each criterion) so instructors know where to find evidence without rewriting level descriptions. A Multi-Eliciting Activity (MEA) is an open-ended assignment that requires the student to use creative methods to solve the problem.

The proposed framework is from the empirical study titled Assessing Cybersecurity Problem-Solving Skills and Creativity of Engineering Students Through Model-Eliciting Activities Using an Analytic Rubric (Kim, et al., 2024) creativity = novelty + usefulness.

The proposed rubric applies to all categories in of the draft universal rubric in development (yara, hunt_query, forensics, report, leadership_deliverable, devops_pipeline, etc.). Documentation and Presentation from the source article are covered by existing critical thinking / execution requirements (methodology, interpretation, required sections); creativity assessment focuses on Originality, Generation & Selection of Ideas, and Value (usefulness).

General 0-4 scale

Levels are described 4 to 0 with distinct differences between each level. The same scale applies to all three criteria. The wording below uses “the submission” and “the procedure” to mean any deliverable (rule, report, code, config, document, analysis) and any described method (methodology, comments, sections, rationale).

Level Label Definition (general)
4 Fully accomplishes The submission/work fully accomplishes the criterion. It is highly unique (originality), or shows strong generation/selection of ideas and extension beyond the minimum, or fully meets exercise needs with clear justification (value); procedure is clear, justified, and complete; rationales for critical choices are provided and correct; no extraneous information.
3 Accomplishes with minor gaps The submission/work accomplishes the criterion. It is unique, involving some new ideas or improvements (they may be predictable or conventional), or shows a tentative attempt to find new uses for common tools or frameworks; meets requirements and is viable; minor embedded errors or rationales that need clarification.
2 Somewhat accomplishes Somewhat accomplishes the criterion. Partially meets requirements; has gaps, errors, or lacks clarity. Does not achieve Level 3.
1 Does not meet minimum Does not achieve Level 2. The work does not meet minimum requirements for the criterion; critical evidence (e.g. results, rationale, or required output) is missing.
0 No evidence / No submission No progress or nothing that resembles a minimal response; no substantive procedure, no submission, or work that does not address the criterion at all.

Three universal criteria and 0-4 level descriptions

The following three criteria are category-agnostic. Instructors apply them to whatever “submission” and “procedure” the exercise produces (YARA rule, hunt query, report, forensics submission, pipeline config, leadership document, etc.). Per-category evidence_location hints (in category_descriptions.md) indicate where to look for evidence without changing these descriptions.

Originality (demonstrates_original_approach)

Criterion: Novelty of the submission or approach-unexpected or inventive use of ideas, structure, or method appropriate to the exercise.

Level Description
4 The submission is highly unique, incorporating novel ideas and displaying inventiveness. Approach, structure, or method is clearly original and justified; rationales for key choices are correct and clear.
3 The submission is unique, involving some new ideas or improvements, although they are predictable or conventional, or shows a tentative attempt to find new uses for common ideas. Approach or structure is explained with minor gaps in rationale.
2 Some non-standard choice or structure but without clear rationale, or one plausible original idea stated but not carried through. Does not achieve Level 3.
1 Purely conventional or template-like response with no stated rationale for approach, structure, or choices. Does not achieve Level 2.
0 No discernible approach, structure, or submission; or submission does not address how the work was done or what was produced.

Generation & selection of ideas / Extends beyond minimum (extends_analysis_with_custom_or_advanced_tooling)

Criterion: Evidence of generating or selecting ideas, methods, or tools beyond the bare minimum (e.g. additional detection features, extra plugins or scripts, richer structure, justified tool or technique choice, or automation that extends depth or reusability).

Level Description
4 Rich extension beyond the minimum: multiple justified choices of methods, tools, structure, or automation; procedure and artifacts align; rationales are correct and complete; serves as a functional solution for the exercise and similar situations.
3 Clear extension beyond the minimum; rationale for choices present with minor gaps; artifacts support the claimed approach; may be conventional but shows deliberate selection or generation of ideas.
2 Some use of extra features, tools, or structure but without clear justification or without extending insight; or extension mentioned but not evidenced in artifacts. Does not achieve Level 3.
1 Only the bare minimum (e.g. minimal required features, no extra tools or structure, no automation or extension). Does not achieve Level 2.
0 No use of required tools/methods demonstrated; or no procedure or artifacts submitted.

Value / Usefulness (provides_enhanced_interpretation_through_technical_depth)

Criterion: Usefulness of the submission for the exercise-thorough, justified interpretation of output or implications; explanation of why findings or choices matter; meeting exercise needs with technical depth and no extraneous or vague content.

Level Description
4 Thorough, well-justified interpretation or explanation; significance of findings or choices clearly tied to objectives; submission meets exercise needs as a functional solution; technical depth where applicable; no extraneous or vague commentary.
3 Clear interpretation or explanation with acceptable rationale; technical depth in explaining significance with minor gaps; meets exercise needs but not fully articulated.
2 Some interpretation or explanation but with little justification; “why it matters” is partial or vague. Does not achieve Level 3.
1 Only minimal interpretation (e.g. one-line labels or no explanation of significance); no clear link to exercise needs or objectives. Does not achieve Level 2.
0 No interpretation of output or implications; raw output or unannotated submission only; no commentary on relevance or meaning.

Creativity Hints for Instructors

All universal rubric categories use the same creativity rubric so that no per-exercise creativity rubric is required. Exercises that assess creativity reference the universal rubric and, when needed, use the evidence_location hints below so instructors know where to look.

Evidence_location hints by category: Where to look for the three criteria (originality, extends_beyond_minimum, value) so instructors can apply the universal level descriptions without exercise-specific text. Use the same three criteria for every category; only the “where to look” changes.

Category Originality (approach, structure, novelty) Extends beyond minimum (tools, methods, features) Value (interpretation, usefulness, why it matters)
yara Rule structure, choice of detection logic, metadata, use of strings/conditions Number and type of detection features, use of modules or advanced syntax beyond minimum Explanation of rule purpose, when it fires, relevance to scenario
hunt_query Query structure, choice of logic (e.g. filters, joins), commented reasoning Extra columns, aggregations, commented logic, use of supported syntax beyond minimal run Explanation of what the query finds, why it matters for the hunt
jupyter_lab Notebook structure, choice of analysis steps, order of cells Cells beyond minimum, visualizations, commented logic, libraries used Interpretation of results, conclusions, relevance to exercise
lab_setup Order of steps, choice of validation method Extra validation or documentation beyond minimum Explanation of what was configured and why it matters
fuzzing Harness design, corpus choice, coverage strategy Harness features, instrumentation, corpus beyond minimum Interpretation of crashes/coverage, relevance to target
static_analysis Analysis path, choice of tools or views (e.g. Ghidra) Exports, annotations, multiple views or scripts Documented findings, significance of artifacts
dynamic_analysis Runtime analysis strategy, choice of instrumentation Tracing options, coverage or behavior logging beyond minimum Documented behavior, interpretation of traces
exploit_code Approach to exploit (e.g. technique, bypass strategy) Code structure, libraries, mitigations Explanation of exploit flow, relevance to objective
report Document structure, section choices, argument flow Sections beyond minimum, methodology detail, sources Findings, recommendations, link to exercise needs
cloud_config Config design, choice of services or settings Extra hardening, least-privilege, documentation Explanation of what was configured and why
red_team_operation Operational approach, TTP selection, planning Payloads, C2, persistence choices; report structure action plan/report; link to objective
pen_test Scan/exploit sequence, tool choice Tools and options used beyond minimum Documented findings, risk/relevance
secure_development Code structure, hardening approach Fixes, libraries, patterns beyond minimum Explanation of fixes and security impact
forensics Analysis path, plugin/step order, hypothesis Plugins, scripts, tooling beyond imageinfo/pslist Annotations, interpretation, link to compromise/timeline
vulnerability_research Research approach, triage method Tools, patch diffing depth, disclosure structure Documented findings, impact, disclosure
capability_development Tool design, choice of approach Code features, options, reproducibility Explanation of what the tool does and when to use it
leadership_deliverable Document structure, metrics/framework choice Sections, metrics, justification depth Link to scenario, stakeholder needs, professional tone
grc_document Document structure, scope, roles Sections, procedures, checklists beyond minimum Clarity of procedures, link to governance/risk
devops_pipeline Pipeline design, stage choice Steps, security checks, config beyond minimum Documentation of steps, build/scan outcome
network_forensics Analysis path, filter/correlation choice Tools, filters, IOC extraction beyond minimum Annotations, interpretation, link to incident
deobfuscation_artifact Recovery strategy, technique identification Recovery method, documentation depth Explanation of obfuscation and recovered artifact
sysadmin_config Config approach, hardening choices Settings, GPO/config beyond minimum Explanation of config and validation
hunt_query_and_report Query + report structure, hunt logic Query features, report sections beyond minimum Findings, methodology, recommendations

Reference

Y. R. Kim, J. Yang, Y. Lee and B. Earwood, Assessing Cybersecurity Problem-Solving Skills and Creativity of Engineering Students Through Model-Eliciting Activities Using an Analytic Rubric, in IEEE Access, vol. 12, pp. 5743-5759, 2024, doi: 10.1109/ACCESS.2023.3348554.

Draft Universal Creativity Rubric

{ “id”: “universal_mea_creativity”, “version”: “1.0”, “description”: “Single MEA-style creativity rubric for all categories. Reference by exercise/category to avoid per-exercise creativity definitions.”, “source”: “Assessing Cybersecurity Problem-Solving Skills and Creativity of Engineering Students Through Model-Eliciting Activities Using an Analytic Rubric (IEEE); creativity = novelty + usefulness.”, “scope”: “All categories in category_descriptions.md. Documentation and Presentation covered by critical thinking/execution; creativity = Originality, Generation & Selection of Ideas, Value.”, “usage”: “Set creativity.use = \”universal_mea_creativity\” (or reference this file). Optional: use evidence_location from category_descriptions.md for where to look.”, “overall_score”: “mean of the three criterion scores (0-4).”, “optional_assessment”: true, “general_scale_0_4”: { “4”: { “label”: “Fully accomplishes”, “definition”: “Product/work fully accomplishes the criterion; highly unique (originality) or strong extension/value; clear, justified, complete; rationales correct; no extraneous information.” }, “3”: { “label”: “Accomplishes with minor gaps”, “definition”: “Accomplishes the criterion; unique but predictable/conventional or tentative new uses for common ideas; viable with minor errors or rationale gaps.” }, “2”: { “label”: “Somewhat accomplishes”, “definition”: “Partially meets requirements; gaps, errors, or lacks clarity. Does not achieve Level 3.” }, “1”: { “label”: “Does not meet minimum”, “definition”: “Does not achieve Level 2; critical evidence (results, rationale, required output) missing.” }, “0”: { “label”: “No evidence / No product”, “definition”: “No progress or nothing that resembles a minimal response; no substantive procedure or submission; work does not address the criterion.” } }, “criteria”: { “demonstrates_original_approach”: { “short_name”: “Originality”, “definition”: “Novelty of the product or approach-unexpected or inventive use of ideas, structure, or method appropriate to the exercise.”, “scale_0_4”: { “4”: “The product is highly unique, incorporating novel ideas and displaying inventiveness. Approach, structure, or method is clearly original and justified; rationales for key choices are correct and clear.”, “3”: “The product is unique, involving some new ideas or improvements (predictable or conventional) or a tentative attempt to find new uses for common ideas. Approach or structure is explained with minor gaps in rationale.”, “2”: “Some non-standard choice or structure but without clear rationale, or one plausible original idea stated but not carried through. Does not achieve Level 3.”, “1”: “Purely conventional or template-like response with no stated rationale for approach, structure, or choices. Does not achieve Level 2.”, “0”: “No discernible approach, structure, or product; or submission does not address how the work was done or what was produced.” } }, “extends_analysis_with_custom_or_advanced_tooling”: { “short_name”: “Generation & selection of ideas / Extends beyond minimum”, “definition”: “Evidence of generating or selecting ideas, methods, or tools beyond the bare minimum-e.g. additional features, extra tools or scripts, richer structure, justified tool choice, or automation.”, “scale_0_4”: { “4”: “Rich extension beyond the minimum: multiple justified choices of methods, tools, structure, or automation; procedure and artifacts align; rationales correct and complete; functional solution for exercise and similar situations.”, “3”: “Clear extension beyond the minimum; rationale for choices present with minor gaps; artifacts support the claimed approach; shows deliberate selection or generation of ideas.”, “2”: “Some use of extra features, tools, or structure without clear justification or without extending insight; or extension mentioned but not evidenced in artifacts. Does not achieve Level 3.”, “1”: “Only the bare minimum (minimal required features, no extra tools or structure, no automation or extension). Does not achieve Level 2.”, “0”: “No use of required tools/methods demonstrated; or no procedure or artifacts submitted.” } }, “provides_enhanced_interpretation_through_technical_depth”: { “short_name”: “Value / Usefulness”, “definition”: “Usefulness for the exercise-thorough, justified interpretation of output or implications; explanation of why findings or choices matter; meeting exercise needs with technical depth.”, “scale_0_4”: { “4”: “Thorough, well-justified interpretation or explanation; significance tied to objectives; product meets exercise needs as functional solution; technical depth where applicable; no extraneous or vague commentary.”, “3”: “Clear interpretation or explanation with acceptable rationale; technical depth in explaining significance with minor gaps; meets exercise needs but not fully articulated.”, “2”: “Some interpretation or explanation with little justification; why it matters is partial or vague. Does not achieve Level 3.”, “1”: “Only minimal interpretation or no explanation of significance; no clear link to exercise needs or objectives. Does not achieve Level 2.”, “0”: “No interpretation of output or implications; raw output or unannotated submission only; no commentary on relevance or meaning.” } } } }

Universal Rubric Applied to an Exercise

Below is an interactive rubric viewer demonstrating how the universal rubric applies to a specific exercise. The example uses the Volatility memory analysis task from the Certified Threat Hunter course. It shows the instructor view with submission requirements, critical thinking criteria, creativity criteria, and video submission checks.

Analyze memory dumps using Volatility

Acquire the memory dump from a target machine and analyze it using Volatility.

Assignment description

Objective

Background

Task

Submission Requirements

Verify these requirements are met before scoring. Unmet requirements may make the submission ineligible for grading.

Critical thinking

Creativity (optional)

Video submission

Confirm the submission meets these requirements.

Grading is for reference only; nothing is saved.